Papers with multi-modal representation
ConceptBert: Concept-Aware Representation for Visual Question Answering (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Visual Question Answering (VQA) is a challenging task that has received increasing attention from both the computer vision and the natural language processing communities. |
| Approach: | They propose an algorithm which learns a joint Concept-Vision-Language embedding for questions which require common sense knowledge from external structured content. |
| Outcome: | The proposed model is based on the Outer Knowledge-VQA and VQA datasets. |
Query Generation for Multimodal Documents (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing approaches to find relevance for multimodal documents with images are expensive and require a lot of runtime overhead. |
| Approach: | They propose to attach generated queries to doc-uments and index them to narrow down to candidate matches using inverted index. |
| Outcome: | The proposed model improves relevance ranking for multimodal documents with images . the proposed model can achieve the state of the art in the first stage retrieval scenarios . |
What Does Your Smile Mean? Jointly Detecting Multi-Modal Sarcasm and Sentiment Using Quantum Probability (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to model multi-modal sarcasm and sentiment are based on quantum probability . sarcasm and feelings embody intrinsic uncertainty of human cognition . |
| Approach: | They propose a quantum probability-driven multi-task learning framework for sarcasm and sentiment recognition using quantum superpositions and quantum interference. |
| Outcome: | The proposed model achieves state-of-the-art in multi-modal sarcasm and sentiment recognition. |
Characterizing the Entities in Harmful Memes: Who is the Hero, the Villain, the Victim? (2023.eacl-main)
Copied to clipboard
Shivam Sharma, Atharva Kulkarni, Tharun Suresh, Himanshi Mathur, Preslav Nakov, Md. Shad Akhtar, Tanmoy Chakraborty
| Challenge: | A common problem associated with meme comprehension lies in detecting the entities referenced and characterizing the role of each of these entities. |
| Approach: | They propose to use a memes dataset on US Politics and Covid-19 memes to characterize the role of harmful entities in memes. |
| Outcome: | The proposed model improves 4% over baseline and 1% over competing models. |
Few-Shot Relation Extraction with Hybrid Visual Evidence (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing few-shot relation extraction methods focus on uni-modal information such as text only. Existing methods focus only on text, requiring only a few labeled instances for training. |
| Approach: | They propose a multi-modal few-shot relation extraction model that leverages both textual and visual semantic information to learn a multiple-modal representation jointly. |
| Outcome: | The proposed model leverages both textual and visual semantic information to learn a multi-modal representation jointly. |